Generalized 3G theorem and application to relativistic stable process on non-smooth open sets

Let G(x, y) and GD(x, y) be the Green functions of rotationally invariant symmetric αstable process in R and in an open set D respectively, where 0 < α < 2. The inequality GD(x, y)GD(y, z)/GD(x, z) ≤ c(G(x, y) + G(y, z)) is a very useful tool in studying (local) Schrödinger operators. When the above inequality is true with c = c(D) ∈ (0,∞), then we say that the 3G theorem holds in D. In this paper, we establish a generalized version of 3G theorem when D is a bounded κfat open set, which includes a bounded John domain. The 3G we consider is of the form GD(x, y)GD(z, w)/GD(x,w), where y may be different from z. When y = z, we recover the usual 3G. The 3G form GD(x, y)GD(z, w)/GD(x,w) appears in non-local Schrödinger operator theory. Using our generalized 3G theorem, we give a concrete class of functions belonging to the nonlocal Kato class, introduced by Chen and Song, on κ-fat open sets. As an application, we discuss relativistic α-stable processes (relativistic Hamiltonian when α = 1) in κ-fat open sets. We identify the Martin boundary and the minimal Martin boundary with the Euclidean boundary for relativistic α-stable processes in κ-fat open sets. Furthermore, we show that relative Fatou type theorem is true for relativistic stable processes in κ-fat open sets. The main results of this paper hold for a large class of symmetric Markov processes, as are illustrated in the last section of this paper. We also discuss the generalized 3G theorem for a large class of symmetric stable Lévy processes. AMS 2000 Mathematics Subject Classification: Primary: 60J45, 60J75 ; Secondary: 31B25, 35J10